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Abstract 
High school freshmen in accelerated courses have known risk and resiliency factors that should 
be considered within systematic efforts to monitor and promote student academic and emotional 
well-being. This study created and evaluated a multi-method approach to identify students in 
Advanced Placement (AP) or International Baccalaureate (IB) courses with signs of risk mid- 
year in terms of stress, affective engagement, and academic performance. A total of 304 ninth 
grade students enrolled in AP/IB coursework and five AP/IB teachers at two public high schools 
in a Southeastern state took part in the screening. Using the researcher-developed screening 
approach, a total of 117 students (38.5%) met criteria for risk in at least one academic or 
emotional area. These results were compared to those obtained using a teacher nomination form, 
which had been developed collaboratively by the teachers and researchers, that specified signs of 
emotional and academic risk. The teacher nomination procedure resulted in the identification of 
39.3% of the at-risk student population (average sensitivity rate = 35.7% across teachers). 
Sensitivity of teacher nominations was higher when identifying academic risk (average = 59.9%) 
as compared to emotional risk (average = 27.9% and 39.6% of students with low school 
satisfaction and high stress, respectively). Findings support the collection of data from students 
(surveys of stress and school satisfaction) and school records (course grades) when identifying 


AP/IB students to consider for targeted services within a multi-tiered system of supports. 


SCREENING STUDENTS IN ACCELERATED COURSES 3 


Identifying High School Freshmen with Signs of Emotional or Academic Risk: 
Screening Methods Appropriate for Students in Accelerated Courses 

To best support students’ academic, behavioral, and social-emotional needs in a 
preventative and systematic manner, schools have increasingly organized service delivery into a 
multi-tiered system of supports (MTSS). MTSS ensures that students are identified early when 
they show any signs of risk and receive services matched to intensity of need. Often referred to 
as population-based school mental health services, MTSS addressing social-emotional needs 
includes prevention, universal screenings to identify students at-risk, and intervention services 
(Doll, Cummings, & Chapla, 2014). Applications of MTSS at the high school level, and tailored 
to students in accelerated curricula (e.g., Advanced Placement, International Baccalaureate) in 
particular, are lacking. This paper describes and evaluates a multi-method approach to 
identifying such students who show signs of academic or emotional risk mid-freshmen year in 
order to provide these students with supports intended to promote emotional well-being and 
success in their accelerated courses. 
Importance of Screening within a Multi-Tiered System of Supports for Mental Health 

Population-based school mental health services are often the only avenue for youth to 
receive needed services for social-emotional concerns. Only one in every four youth in the U.S. 
with a diagnosable psychiatric disorder reports receiving services within community or clinic- 
based settings (Merikangas et al., 2010). Symptomatic youth are more likely to turn to school- 
based rather than community-based services (Slade, 2002). School settings are a particularly 
feasible setting to offer preventive or early intervention services for youth who are experiencing 
challenges with typical life transitions and/or signs of mental health problems that, if left 


untreated, may cause significant distress and necessitate more intensive intervention. 
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Preventative mental health treatments for at-risk adolescents, utilizing time-limited, evidence- 
based interventions, can reduce symptomology for students targeted for elevated levels of 
depression (Brent et al., 2015; Rodhe, Stice, Shaw, & Gau, 2015), trauma (Stein, Jaycox, 
Kataoka et al., 2003) and externalizing behaviors (Simonsen, Myers, & Briere, 2011). Such 
targeted interventions use screening procedures to identify students displaying elevated risk 
factors or symptoms and are in need of additional supports. When determining which students 
may benefit most from prevention services, schools can conduct universal screenings — 
structured assessment of all students in a given population — to guide decision-making for which 
students need more intensive assessment and treatments (Albers & Kettler, 2014). 
Screening for Risk among High School Students in Accelerated Curricula 

School-based mental health models that use a MTSS approach are designed to meet the 
unique needs of specialized populations (Christner, Mennuti, & Whitaker, 2009). One population 
that has received less attention in terms of school mental health services is students in Advanced 
Placement (AP) classes or International Baccalaureate (IB) programs (hereafter referred to as 
“AP/IB”). AP/IB are becoming increasingly widespread accelerated curricular options in high 
school settings (Bunnell, 2011; Doherty, 2009; Jeong, 2009; Spalding, Eden, & Heppner; 2012). 
AP/IB coursework provides students identified as high-achieving or gifted with appropriately- 
challenging curricula that may be credited by institutions of higher learning for college credit 
(see College Board, 2018, and IBO, 2018, for program details). 

Students in AP/IB have likely been understudied when it comes to school mental health 
services because their typical history of academic achievement has been equated with less of a 
need for social-emotional supports (Suldo, Gormley, DuPaul, & Anderson-Butcher, 2014). 


However, this population’s frequency of mental health problems — such as diminished life 
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satisfaction and elevated psychopathology symptoms — is on par with levels documented among 
general samples of youth in the U.S. (Suldo, Shaunessy-Dedrick, Ferron, & Dedrick, 2018). 
Comparisons of mean levels of mental health among subgroups of students in (a) AP classes, (b) 
the IB program, or (c) general education indicate that AP/IB students manifest levels of 
internalizing problems that are similar to their peers in general education, as indexed by 
comprehensive measures of psychopathology or narrowband measures of anxiety (Suldo & 
Shaunessy-Dedrick, 2013a). Thus, despite the academic success that led to their pursuit of 
college-level coursework, AP/IB students are likely to have typical rates of need for mental 
health services, albeit services tailored to their particular experiences in accelerated curricula. 

A unique risk factor that differentiates AP/IB students from their peers in general 
education is the heighted risk of experiencing stress due to school-related challenges inherent to 
their rigorous curricular expectations (Neihart et al., 2002; Suldo, Shaunessy, & Hardesty, 2008; 
Suldo & Shaunessy-Dedrick, 2013a). High levels of stress can have a cascading negative effect, 
as elevated stress predisposes youth to turn to ineffective coping responses such as rumination, 
which puts youth at higher risk for psychopathology (Compas, Orosan, & Grant, 1993). A 
synthesis of prospective studies found robust support that elevated stress predicted increases in 
psychopathology, particularly internalizing problems (Grant, Compas, Thurm, McMahon, & 
Gipson, 2004). For students in elite college preparatory schools, a population that shares some 
characteristics with youth in accelerated high school curricula, increasing levels of stress co- 
occur with decreasing levels of life satisfaction (Feld & Shusterman, 2015) and higher levels of 
psychopathology (Suldo et al., 2008). The latter study of IB students revealed considerable 
overlap between self-reported perceived stress and internalizing symptoms of psychopathology 


(r = .72; whereas r = .40 with externalizing problems). 
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In order to develop a screening tool that efficiently and accurately identifies AP/IB 
students who are at-risk, it is important to examine factors that are associated with AP/IB student 
success. Recent research of over 2300 AP/IB students explored resiliency factors relevant to the 
academic and emotional success of students in a high-stress curricular context (Suldo et al., 
2018). In addition to highlighting the importance of specific coping strategies students use to 
manage their academic demands, Suldo et al. (2018) found higher levels of achievement 
motivation (i.e., drive to learn and achieve) and student engagement (cognitive, behavioral, and 
affective) predicted more positive mental health and academic outcomes. In particular, 
motivation and affective engagement (i.e., satisfaction with one’s AP/IB program combined with 
positive feelings about one’s school and teachers) yielded significant, unique effects on higher 
life satisfaction, lower psychopathology, and lower school burnout. Using a motivational 
conceptualization of classroom engagement and disaffection, affective/emotional engagement is 
reflected in experiences of positive emotions such as enthusiasm, interest, enjoyment, 
satisfaction, and pride (Skinner, Furrer, Marchand, & Kindermann, 2008). To that end, school 
satisfaction, which reflects feelings of happiness in the school domain, may be a valuable 
indicator of belongingness and affective engagement. The value of school satisfaction is 
supported by longitudinal studies that document significant prospective relationships with 
adolescents’ mental health including social-emotional strengths and symptoms of internalizing 
problems (Lester, Waters, & Cross, 2013; Moffa, Dowdy, & Furlong, 2016). 

In sum, perceived stress and affective engagement are factors that have emerged as 
salient to AP/IB students’ success, especially in the emotional health domain. One way to 
identify AP/IB students at-risk for diminished emotional and academic success is through direct 


assessments of those outcomes (e.g., clinical levels of mental health problems, academic failure 
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as indicated by withdrawal from AP/IB programming). A more preventative approach could 
involve assessment of the most salient risk factors: elevated stress, low affective engagement, 
and/or early signs of struggles in courses as indicated by below par grades in the first semester of 
high school. Armed with this knowledge of the unique risk factors for AP/IB student success, 
educators can create specialized screening procedures for this growing high school population. 
Common Sources of Information in School-Based Screening 

When designing universal screening procedures for any population, schools have several 
methods to consider. Each method has strengths and limitations that pertain to accuracy and 
feasibility. Regarding the latter, the least invasive and readily available method involves data 
housed in school records, such as review of office discipline referrals (ODRs) to identify 
students in need of additional services. However, AP/IB students experience far fewer ODRs and 
school absences than high school students in general education (Suldo & Shaunessy-Dedrick, 
2013a). Given the importance of a relatively high grade point average (GPA; e.g., > 3.0) to 
students’ initial placement in accelerated classes, the more appropriate school records to access 
may be grades earned in 9th grade AP/IB classes. Students with higher ability and achievement 
levels tend to be especially accurate reporters of their course grades (Schwartz & Beaver, 2015). 

Another commonly used identification method is referrals from concerned teachers, 
parents, and students. Referral methods are natural avenues for students to be recommended for 
participation in special education and to receive other supports, such as social-emotional 
interventions. Limitations include the reactive nature of when referrals are ultimately triggered, 
and systematic errors in accuracy. Eklund and Dowdy (2014) found that students who have 
higher academic performance are missed in referral methods more often than when rating scales 


are used in screening procedures to identify at-risk students. Such bias against high-achieving 
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youth suggests rating scales may be preferable to referrals when the target population involves 
students in accelerated courses. 

Rating scales completed by students or informants (e.g., teachers, parents) afford a 
systematic look at the functioning of all students in a target group to determine whom should be 
connected to services. The most appropriate source(s) of information on behavior varies by 
student age. Parents often are preferred raters of the behavior of younger students. With regard to 
adolescents, students are generally judged to be the most accurate and reliable informant of 
multiple forms of mental health problems, including internalizing, covert externalizing, and 
substance use behaviors (Levitt, Saka, Romanelli, & Hoagwood, 2007). Particularly during the 
middle and high school years, when teachers serve multiple section of classes (thereby reducing 
familiarity with a typical student and increasing the time needed to rate all students) and youth 
may be less likely to spontaneously disclose mental health difficulties to adults, youth appear the 
best option to provide ratings of their own emotional status (Raines, Dever, Kamphaus & Roach, 
2012). Rating scales often flag more students as at-risk compared to ODRs and teacher 
nomination methods (Miller et al., 2015), likely leading to the identification of more high- 
achieving students who could be missed by traditional school referral methods (Eklund & 
Dowdy, 2014). Challenges to incorporating rating scales in screenings pertain to logistical and 
ethical issues to be navigated in order to gather, score, and interpret potentially sensitive data 
from a large number of stakeholders. In addition to securing parental consent and valid ratings 
for each eligible student, educators must determine a cut score that appropriately designates a 
student as showing significant risk beyond the norm seen in the target population, but ideally 
does not identify more students as at-risk than can be supported by available mental health staff 


(i.e., a serviceable base rate; Kilgus & Eklund, 2016). 
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Teacher nominations represent a screening procedure that is arguably less time-intensive 
than gathering supplemental ratings for each student; instead, teachers systematically consider all 
of their eligible students, and ‘nominate’ or identify which student(s) they believe are at-risk on 
the basis of demonstrating a given set of social-emotional-behavioral symptoms. As an example 
of efficiency, the Systematic Screening for Behavior Disorders (SSBD-2; Walker, Severson, & 
Feil, 2014) starts with a teacher nomination procedure within a multiple-gating screening 
procedure whereby teachers complete rating scales for only the top three students nominated as 
at-risk for internalizing or externalizing problems. Strengths of teacher nominations involve the 
low cost, ready access to teachers, and potential accuracy, as teachers are assumed to be valid 
informants of aberrant behavior given their familiarity with large samples of youth in an 
academic and social setting. Alas, the accuracy of teacher identification is far from perfect with 
regard to emotional risk; only a quarter to half of students with internalizing problems (verified 
via repeat self-report of symptoms or through diagnostic interview) are correctly identified as 
such in nomination procedures (Auger, 2004; Cunningham & Suldo, 2014; Moor et al., 2007). It 
is possible that improved accuracy follows the specification of clear manifestations of risk being 
made available to the teacher participants. On the other hand, it may be the case that a sizeable 
proportion of students with internalizing behavior will often fall under the teacher’s radar due to 
students’ concealment of obvious symptoms or lack of awareness by busy educators without 
psychological training. 

Screening for internalizing problems. When identifying students with emotional 
concerns, some of the more easily accessed screening methods such as review of ODR data and 
referrals from educators may result in an over-identification of students with externalizing 


concerns and under-identification of students with internalizing symptoms (Splett, George, et al., 
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2018). In a discussion of MTSS for internalizing forms of mental health problems, Weist and 
colleagues (2018) underscore the need for early, targeted interventions (i.e., Tier 2 supports) for 
students with specific risk factors (for instance, elevated stress) or “who screen positive on 
indicators that suggest risk of internalizing concerns but for whom functioning is not yet 
significantly impaired” (p. 178). When comparing students identified by traditional school 
referral methods to peers identified through use of teacher rating scales to gather data on all 
students, students identified through universal screening but missed by school referral tended to 
have less severe levels of symptoms and less academic risk (Splett, Trainor, et al., 2018). Thus, 
rating scales appear especially relevant to identifying students with moderate levels of 
internalizing symptoms, in particular those without academic deficits. 

There are a handful of student self-report rating scales that are relatively brief (i.e., < 25 
items) and measure internalizing symptoms, including the Behavioral Assessment System for 
Children, Third Edition- Behavioral and Emotional Screening System (BASC-3 BESS; Reynolds 
& Kamphaus, 2015), the Behavior Intervention Monitoring and Assessment System (BIMAS; 
McDougal, Bardos, & Meier, 2011) and the Strengths and Difficulties Questionnaire (SDQ; 
Goodman 1997). The SDQ is available for free in the public domain, whereas the BESS and 
BIMAS are commercially available and require schools to pay per rating form. Although the 
number of psychometrically sound screening measures is growing, existing omnibus rating 
scales of symptoms do not necessarily assess the risk factors most salient for AP/IB youth (stress 
and disaffection with school) or match the targeted supports most appropriate for that population. 
For AP/IB students, interventions focused on stress management and student engagement may be 
most appropriate, given the population’s unique risk and protective factors. 


Aims of Current Study 
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The present project aimed to create a multi-method, usable, and accessible approach to 
identifying AP/IB ninth grade students who were at-risk for academic or emotional challenges, 
with the intent to provide Tier 2 supports focused on stress management and school engagement 
during students’ second semester of high school. Research questions included: 

1. What is the proportion of AP/IB students in 9th grade that demonstrate risk on factors 
most salient to AP/IB student success—perceived stress, affective engagement, and 
academic performance—as indicated by student report of emotional factors and school 
records of grades earned in first semester classes? 

2. How accurate are teacher nominations in identifying those students who demonstrated 
risk on stress or affective engagement per self-report, or academic performance per 
school records? 

3. How accurate is student self-report of first semester course grades in identifying those 
students who demonstrated risk on academic performance per school records? 

In the first semester of high school, the sample examined in this study took part in a research- 
based Tier | intervention to build skills in engagement and coping with academic stress through 
a 10-12 lesson social-emotional learning (SEL) program delivered weekly through students’ IB 
Inquiry Skills or AP Human Geography class. Given their participation in a universal support 
tailored to their curricular experiences, we hypothesized that 15-20% of students would 
demonstrate risk on the emotional factors targeted in the SEL program in line with the notion 
that core instruction is expected to meet the needs of about 80% of students (NASDE, 2005). 
Similarly, we hypothesized that 15-20% of students would be at risk academically, which is 
lower than the 24% identification rate for academic risk (i.e., GPA < 3.0) yielded in prior 


research with a large sample of AP/IB students in grades 9-12 without such systematic supports 
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(Suldo et al., 2018). Given the internalizing nature of the targets, we hypothesized teacher 
nominations would be less accurate in identifying students with emotional risk (e.g., miss rates > 
50%; Auger, 2004; Moor et al., 2007), compared to identifying students with academic risk 
given their knowledge of academic performance in at least their class. Most students in AP/IB 
are acutely aware of their academic performance given the importance of course grades for entry 
and retention in AP/IB and the salience of GPA to students’ future goals, like college admission. 
Thus, we hypothesized that students would be highly accurate reporters of their grades. 
Method 

Sample 

Participants come from two public high schools (hereafter referred to as School A and 
School B) from a university-district partnership in a Southeastern state. Details on the School 
Directory Information of the National Center for Educational Statistics (NCES) indicate that 
School A is in a large city locale, serving 1,639 students (52.0% female; 43.9% eligible for free 
or reduced-price lunch; 46.8% White, 22.8% Hispanic, 14.5% Black, 7.4% Asian, 8.2% 
multiracial, <1% Native Hawaiian/Pacific Islander, <1% American Indian/Alaska Native). 
School B is in a large suburb locale, serving 2,355 students (51.3% female; 21.2% eligible for 
free or reduced-price lunch; 63.1% White, 23.6% Hispanic, 6.3 % Black, 2.4% Asian, 4.2% 
multiracial, <1% Native Hawaitian/Pacific Islander, <1% American Indian/Alaska Native). 

Students. There were 495 students in 9th grade in School A; 163 of these students in the 
IB Inquiry Skills course were the student population of interest in this study. Of the 613 students 
in 9th grade in School B, the 193 taking AP Human Geography were targeted for this study. In 
total 356 AP/IB 9th grade students were targeted; parent consent to participate in the evaluation 


of the larger MTSS was received for 332 students (93.3%) in August 2016. By the mid-year 
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screening in January 2017, 10 students had left the IB program and 3 students dropped the AP 
Human Geography class. Parents of the remaining 319 AP/IB participants received a 
“notification of screening” letter explaining that AP/IB freshmen would complete a short survey 
on stress and feelings about school and that students’ ratings would be part of the data reviewed 
to determine which students would be offered Tier 2 support. Parents were asked to return the 
form within one week if they wished to refuse their child’s participation in the screening. 
Thirteen (11 from School A, 2 from School B) of 319 students (4.1%) were excluded from the 
screening due to return of the parent notification form. 

Two students were absent during the screening period, resulting in a final sample of 304 
students (59.2% female; 53.3% White, 23.0% Hispanic, 2.6% Black, 7.9% Asian, 7.9% 
multiracial, <1% Native Hawatian/Pacific Islander, <1% American Indian/Alaska Native, 1% 
other ethnic identity, 3.3% unknown [did not report]; 43.8% IB [n = 133 students], 56.3% AP [n 
= 171]). Of note, accelerated programs were initially established to challenge advanced students 
and continue to be recognized as an appropriate high school curriculum for gifted and high 
achieving students (Colangelo, Assouline, & Gross, 2004). The demographic features of our 
sample are consistent with other research findings indicating that Black students in particular are 
underrepresented in AP and IB (Kolluri, 2018; Wildhagen, 2014). Parent educational attainment 
was examined as an indicator of SES; 74.4% and 82.9% of mothers and fathers, respectively, had 
a college degree or beyond, suggesting a relatively affluent sample. No data on eligibility for 
free or reduced-price lunch were collected from these youth. 

Teachers. Five teachers took part in this study. The three teachers from School A taught 
a total of seven sections of IB Inquiry Skills, and two teachers from School B taught a total of 


nine sections of AP Human Geography. Three teachers were male (60% of teacher sample); four 


SCREENING STUDENTS IN ACCELERATED COURSES 14 


were Caucasian (80%) and the fifth was Hispanic (20%). On average, the participants had been 
teaching high school for 12 years (range = 8 — 20 years). To maximize the likelihood of teacher 
sensitivity to student risk, we involved teachers who (a) observed their students participate in the 
Tier 1 intervention, (b) collaborated in the development of the nomination form, and (c) 
understood the screening data would be used to select students for Tier 2 supports. 
Procedure 

The screening was embedded within a MTSS implemented to support AP/IB freshmen in 
the two partner schools. The supports were developed through an iterative collaboration between 
university-based researchers (manuscript authors) and the district from which the two schools 
were drawn. In the university-school partnership, faculty in school psychology and gifted 
education and school psychology trainees partnered with district educators to design and carry 
out the universal and selective supports for AP/IB freshmen. 

Tier 1. All 356 freshmen enrolled in AP/IB at the start of the school year were exposed to 
a SEL program as part of their school’s commitment to supporting AP/IB students during the 
transition to high school. This universal support provided in the fall semester (September- 
December 2016) entailed a 10 to 12 lesson, research-based SEL program delivered weekly by 
university-based interventionists during AP Human Geography or IB Inquiry Skills. The new 
curriculum—referred to as the Advancing Coping and Engagement (ACE) program (for a 
description, see [authors, in press])— involved direct instruction in, and rehearsal of, skills 
pertinent to coping (e.g., responding to school stress through time and task management 
strategies and cognitive reappraisal) and engagement (e.g., forming relationships with teachers, 
generating positive feelings about one’s school and academic program). These topics were 


emphasized because they are skills AP/IB students can apply across their high school 
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coursework and address targets empirically linked to the academic and emotional success of 
AP/IB students (Suldo et al., 2018). This service modality mimics other types of Tier 1 mental 
health supports in that it resulted from school-community teaming to select lessons from a 
manualized SEL curriculum that can be integrated into classwide instruction to provide a strong 
foundation for all students (Weist et al., 2018). Classroom teachers are the intended end-users of 
the ACE program. The five teachers were present for the majority of lessons, and assisted 
primarily with behavior management and occasionally with comments relevant to lesson content. 
Tier 2. The mid-year screening process (described below) identified youth in need of 
additional supports based on mid-year review of student academic and emotional well-being 
data. The Tier 2 intervention was provided in the spring semester (February-April 2017), and 
entailed 1-2 individual “coaching” meetings with the university-based interventionist. This 
service modality mimics other types of Tier 2 supports for students at-risk for internalizing 
problems in that it is targeted, time-limited, and builds upon skills that are taught in Tier 1 and 
relevant to student mental health (Weist et al., 2018). This new Tier 2 intervention—teferred to 
as Motivation, Assessment, and Planning (MAP) meetings (for a description, see [authors, in 
press])—1is unique in that it was specifically developed to target the academic and emotional 
stressors (e.g., low school satisfaction, high level of stress, subpar grades) faced by AP/IB 
students. Using a motivational interviewing approach, students reflect on their current levels of 
coping and engagement as compared to a large normative sample of AP/IB participants, identify 
targets for improvement, and create an action plan to ultimately lessen their stress and improve 
emotional well-being and/or academic success in AP/IB and beyond. School psychologists, 
school counselors, and other qualified school mental health professionals are the intended end- 


users of the MAP meetings. 
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Screening. In January 2017, 304 participants completed the 1-page paper-and-pencil 16- 
item screening survey (measures of stress, school satisfaction, and fall academic performance, 
described below) during their AP/IB class. The introduction and administration of the survey 
spanned 5-10 minutes. Survey data were entered into an Excel spreadsheet during the same 
school day. Researchers returned to ask some students to respond to skipped items as needed and 
to administer the survey to previously absent students. 

During the same week that student survey data were collected, a research team member 
met with the classroom teacher to collect teacher nomination data. The researcher first explained 
the purpose of the screening: 

To identify students who at mid-year, show or report signs of academic or emotional risk 

in AP/IB and thus may benefit from brief, individualized support to address academic or 

emotional challenges in AP/IB. Academic risk: GPA < 3.0, grades of C or lower in 

AP/IB classes. Emotional risk: elevated stress, negative feelings about schooling 

experiences. 

The handout summarized the supports offered through the Tier 2 intervention and noted that the 
eligibility process involved a review of data from student report of emotional health, academic 
records, and educator nominations. 

Teachers were then given the three-page nomination form. The first page included 
instructions for use and descriptions of 14 behavioral examples of academic or emotional 
challenges common among AP/IB youth (see Appendix for full list). On the bottom of the form, 
teachers could write in additional signs of student risk they considered while nominating 
students; no teachers utilized this option. The second page included an example nomination form 


with hypothetical student names to illustrate a completed Educator Identification form. The third 
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page provided a roster listing eligible students in the teacher’s AP/IB class. By each student’s 
name teachers were asked to check yes (student shows signs of risk) or no (student does not 
show signs of risk) based on the 14 behavioral examples provided on page one. Teachers also 
had the option to indicate if they did “not know student well enough to judge” the student’s risk 
status (e.g., student is new to class), but no teacher selected this option. Teachers completed the 
nomination form for a given class section while the research team administered the student 
surveys. In one instance, a researcher returned to the teacher who marked “yes” for few students 
(3 of 72) and encouraged her to reconsider her roster with the aim of identifying additional 
students who showed any of the signs listed on the first page. The teacher examined her roster 
but did not change her original responses, noting that any additional students who she felt 
demonstrated any signs of academic or emotional challenges had already withdrawn from the IB 
program, or did not have permission to participate in the screening (n = | student); thus, those 
students names did not appear on the list of students eligible to nominate. 
Measures and Indicators Used in the Screening 

Perceived Stress Scale (PSS; Cohen, Kamarck, & Mermelstein, 1983). Students 
completed a six-item version of the PSS that included only the non reverse-scored items, so that 
all items indicated greater stress. Respondents indicated how often in the last month they 
experienced various aspects of stress, using a five-point response scale: (1) Never, (2) Almost 
Never, (3) Sometimes, (4) Fairly Often, and (5) Very Often. A sample question from the PSS 
states, “In the last month, how often have you felt difficulties were piling up so high that you 
could not overcome them?” The PSS has yielded excellent internal reliability in prior research 
with AP/IB students (a = .91; Suldo et al., 2008), as well as among samples of adolescents not 


drawn from specialized curricula (a = .82 - .88; Bluth, Campo, Futch, & Gaylord, 2017; Galla, 
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2016). Support for construct validity is provided by a strong relationship between PSS scores and 
frequency of stress related to academic requirements (r = .53; Suldo et al., 2015), increased 
depressive symptomatology (r = .68; Galla, 2016), and reduced life satisfaction (r = -.54; Galla, 
2016). In the current study, a = .85 for the PSS. 

School Satisfaction scale of the Multidimensional Student Life Satisfaction Scale 
(MSLSS; Huebner, 1994). The MSLSS is a self-report measure of life satisfaction in different 
domains, including family, friends, living environment, self, and school. The school satisfaction 
scale, consisting of eight items, such as “I look forward to going to school,” was used as part of 
the screening procedures. Respondents indicate how over the past several weeks they have felt 
and thought about school using a six-point response scale: (1) Strongly Disagree, (2) Disagree, 
(3) Mildly Disagree, (4) Mildly Agree, (5) Agree, and (6) Strongly Agree. The school satisfaction 
scale has been used in previous studies as a proxy of student school belonging (Moffa, Dowdy, 
& Furlong, 2016) and yielded high internal consistency (a = .84) and inverse correlations with 
internalizing problems among secondary students (Gini, Marino, Pozzoli, & Holt, 2018). 
Regarding construct validity, Huebner (1994) reported a strong, positive relationship (r = .68) 
between school satisfaction scores and the Quality of School Life Scale (Epstein & McPartland, 
1976). In the current study, a = .86 for school satisfaction. 

Teacher nomination form. The nomination form was created one month before data 
collection through a collaboration between the university-based research team (which included 
experts in the fields of school psychology, gifted education, measurement design, and 
educational statistics), the assistant principal, and school psychologist at each school, and the 
five AP/IB teachers. Researchers held a total of four focus groups across the two school sites. At 


each site, one focus group was held with teachers and one focus group was held with the 
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assistant principal responsible for AP/IB and the school psychologist. At the start of each focus 
group, the researchers described the upcoming selective intervention goal (1.e., help students 
reflect on and further develop healthy coping and engagement practices that are linked to 
emotional and academic success in AP/IB courses), intended population (i.e., students who, mid- 
year, show or report signs of academic or emotional challenges in AP/IB), and the eligibility 
process (i.e., screening with student self-report of emotional health, review of academic records, 
and educator nominations). Focus group participants were asked “When you think about your 
current students who need or may most benefit from additional support, what characterizes these 
students, in terms of academic performance, emotional health, or concerns expressed by parents 
or classmates?” Educators reflected upon their experiences, recalling the behaviors demonstrated 
by students they taught previously who ultimately earned failing grades, dropped the course, 
and/or reported considerable emotional distress during the course. 

After a discussion of risk signs, the researchers presented a draft nomination form that 
included eight signs of emotional challenges related to school-related stress. Signs of challenges 
included features reflecting elevated stress levels (e.g., Difficulty coping effectively with 
academic demands), poor student engagement (e.g., Makes negative statements about AP/IB or 
school), as well as overt manifestations of academic problems (e.g., Poor test, quiz, and exam 
grades) or emotional challenges (e.g., Seems unhappy during class). To verify accuracy of the 
risk signs in the first draft of the education nomination form, participants provided additional 
feedback to the clarity and relevance of these signs and offered additional ways that students 
appear at-risk for emotional or academic challenges. This discussion generated six more signs 
indicative of problematic responses to stress (e.g., Gives up or stops trying on schoolwork), 


academic engagement (e.g., Does not seem to take schoolwork seriously), emotional health (e.g., 
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Expresses extreme or frequent worry about performance on assignments or exams), and 
academic performance (e.g., Misses class). Some of the original eight signs were modified to be 
more descriptive. For instance, “tearful” was added to “Seems unhappy during class.” Other 
signs were reworded for relevance to the classroom. For instance, “Avoids or withdraws from 
social situations” was changed to “Appears lonely or socially isolated (no friends in AP/IB).” 
There was considerable overlap between focus groups with regard to signs generated, with the 
final group indicating that the compiled list of 14 signs (see Appendix) developed during the 
focus group meetings was exhaustive. 

Academic performance. School administrators accessed student participants’ fall 2016 
report cards and provided the research team with students’ unweighted fall semester GPA, and 
the grade earned in IB Biology (School A) or AP Human Geography (School B). School 
administrators indicated IB Biology would be a better indicator of academic risk than the grade 
earned in IB Inquiry Skills because of the larger distribution of grades in the Biology course. The 
Inquiry Skills class typically produced a grade distribution with most grades being A or B. In 
regular grade-level wide meetings in which student performance in the IB program was 
discussed, teachers learned of their students’ progress in other IB course, supporting the assumed 
familiarity of Inquiry Skills teachers with students’ performance in Biology and other courses. 

The task of gathering end-of-course grade information from school records and merging 
data from school records with data from student ratings of emotional well-being might be 
supplanted by reliance on student self-report of academic performance if the latter is highly 
accurate. To examine agreement between data from school records and student self-report, 
students were asked about the grades they earned during the first semester of 9th grade. At the 


bottom of the 1-page form that included the PSS and MSLSS school satisfaction items, students 


SCREENING STUDENTS IN ACCELERATED COURSES 21 


answered two questions: “What was your unweighted GPA from fall 2016 (e.g., 3.25)?” and 
“What grade did you earn in AP Human Geography [IB Biology]?” 
Overview of Analyses 

Students were dichotomized into two groups (at-risk or not) in each of three domains: 
stress, engagement, and academic performance. At-risk due to high perceived stress was defined 
as PSS score > 3.6. Low affective engagement (school satisfaction) corresponded to a MSLSS 
score < 3.4. At-risk due to academic performance was based on semester GPA and grade earned 
in AP Human Geography or IB Biology. The former estimates overall progress across courses, 
and the latter indicates performance in rigorous accelerated coursework in particular. Given the 
high correlation between these variables (7 = .76), we created a combined academic risk status 
variable. Students were dichotomized into two academic risk groups: (1) “at-risk academically” 
= unweighted fall semester GPA < 3.0 or grade of C, D, or F in designated AP/IB course, or (2) 
“not at-risk academically” = GPA > 3.0 and grade of A or B in AP/IB course. 

Although these cut scores for academic and emotional risk might seem atypical for a 
general sample of high school students, they were selected for conceptual and empirical reasons. 
In the current sample, 15-16% of students were identified as at-risk on each indicator, which is 
similar to a T score of 60 (one SD above the sample mean). For the PSS (scores ranging from 1- 
5), a response option of 3 corresponds to “Sometimes” and a 4 to “fairly often”; a mean score 
above 3.6 is closer to experiencing stress frequently rather than infrequently. This cut score is 
also higher than the average PSS score reported mid-year by previous samples of high school 
students in AP or IB (e.g., M = 2.77 to 3.09) and general education (M = 2.57 to 2.75; Suldo & 
Shaunessy-Dedrick, 2013a; Suldo & Shaunessy-Dedrick, 2013b). 


On the MSLSS (scores ranging from 1-6), responses of | (strongly disagree) — 3 (mildly 
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disagree) indicate that a student disagrees with a statement reflecting positive feelings about 
school; a mean score below 3.4 is closer to dissatisfaction with school as opposed to satisfaction 
with school. This cut score is also lower than the average school satisfaction score reported mid- 
year by previous samples of high school students in AP or IB (e.g., M = 3.93 to 4.13) and general 
education (M = 3.80 to 4.00; Shaunessy, Suldo, Hardesty, & Shaffer, 2006; Suldo & Shaunessy- 
Dedrick, 2013a). Our cut scores for elevated levels of stress, low satisfaction with school, and 
below average course grades match the signs of underachievement cited in the literature on 
gifted youth, which includes students taking AP/IB classes. Gifted students who are identified as 
“underachievers” often report negative attitudes towards school and their teachers, low academic 
self-perceptions, and low self-regulatory skills (Siegle & McCoach, 2018). 

Behavioral indicators of gifted underachievement can include failure to complete 
assignments or produce high quality work, which results in lower course grades than expected 
for a student of their abilities (McCall, 1994; Siegle & McCoach, 2018). Educators and parents 
of high achieving youth similarly report lack of motivation, poor academic self-perceptions, and 
low work completion among their top concerns for student’s behavior at school and at home 
(Siegle & McCoach, 2018). The academic cut scores also had good face validity among our 
sample of AP/IB educators, families, and students, who shared that grades of A or B are 
generally expected in accord with perquisite performance needed for permission to take more 
college-level classes in high school, and to ultimately be competitive for college admission as 
well as earn the IB diploma. To illustrate, in a large, diverse sample of AP/IB students in grades 
9 — 12, the mean unweighted GPA for a semester of courses was 3.29 (Suldo et al., 2018). 

A composite risk variable was created that differentiated students with any risk (at-risk 


levels of stress, engagement, academic performance, or a combination of 2 or 3 risk factors) from 
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those students with no risk factors. This dichotomized risk composite variable was compared to 
teacher nomination status (“yes” or “no” at-risk for diminished success in AP/IB). Accuracy of 
teacher nominations was evaluated in terms of specificity and sensitivity. Diagnostic efficiency 
statistics (Landau, Milich, & Widiger, 1991) were calculated with regard to true positive rate 
(percentage of students identified with risk based on grades earned per report cards or self-report 
of stress or school satisfaction, and correctly identified as at-risk by teachers), false negative rate 
(proportion of at-risk students missed by teacher nominations), false positive rate (percentage of 
students without risk per survey and school records, but teachers misidentify as at-risk), and true 
negative rate (proportion of students without risk per data from surveys or school records, that 
teachers did not nominate as at-risk). Additional estimates were made for positive predictive 
value (percent of the teacher nominated students who were identified with risk based on grades 
and self-report) and negative predictive value (percent of the students not nominated by the 
teacher who were not identified with risk based on grades and self-report). 

The proportion of true positives and proportion of false negatives for each area of risk 
(i.e., stress, engagement, academic performance) was then examined. For students at risk in a 
particular area (e.g., stress), we determined the proportion identified as at risk by the teacher 
(true positives) and the proportion missed by the teacher (false negatives). Because teachers 
made nominations for students at risk for diminished success in AP/IB and did not make specific 
nominations for risk of stress, engagement, and academic performance, we were not able to 
estimate area specific (i.e., stress, engagement, academic performance) true negative or false 
positive rates. 

In addition to computing the diagnostic efficiency statistics for each teacher, differences 


between teachers were tested using Pearson’s y2, and across-teacher average values were 
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obtained. These average values, as well as corresponding confidence intervals, were obtained by 
using generalized mixed linear models with a binary probability distribution, a logit link 
function, a random effect for teacher, and Kenward-Roger degrees of freedom (to account for the 
small sample size). 
Results 

Descriptive Statistics for Indicators in Screening 

The sample evidenced considerable diversity on academic and mental health measures. 
Regarding the latter, the mean score for school satisfaction was 4.29 (SD = 0.84; range = 1.88 to 
6.00), and the mean level of perceived stress was 2.67 (SD = 0.87; range = 1.00 to 5.00). These 
mean values are comparable to the average scores seen in research with other samples of AP/IB 
students (Suldo & Shaunessy-Dedrick, 2013a; 2013b). On academic outcomes, the mean GPA 
for courses completed in the fall 2016 semester was 3.57 (SD = 0.38; range = 2.31 to 4.00). A 
closer look at grades earned in either AP Human Geography or IB Biology revealed that the 
average grade was in the B range (M = 3.19, SD = 0.78; range = 0 to 4.00) and the distribution 
was as follows: A (39.5%), B (42.4%), C (16.5%), D (1.3%), and F (0.3%). 

The mean number of eligible students presented per teacher on the educator nomination 
form was 61 (SD = 26.5) and ranged from 20 students (i.e., 1 section of AP/IB freshmen) to 93 
students (i.e., 5 sections of AP/IB freshmen). Teachers nominated an average of 26.3% of 
students on their roster(s) as at-risk; percentage nominated ranged between 4.2% (1.e., teacher 
who nominated 3 of 72 students in her 4 sections) and 40.0% (1.e., teacher who nominated 8 of 
20 students in his 1 section) for a given teacher. A total of 76 students (25.0% of the sample) 
were nominated by teachers as at-risk. 


Prevalence of Risk among AP/IB Freshmen 
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Using the aforementioned cut scores, 117 of the 304 students (38.5%) screened had risk 
in at least one area (stress, engagement, or academic performance). Most of students identified as 
at-risk had elevations in only one area (n = 84 with 1 risk factor, n = 27 with 2 risk factors, and n 
= 6 with 3 risk factors). The proportion of students identified as at-risk in each area is presented 
in Table 1. The small overlap between risk indicators is notable. A minority of students who met 
criteria for emotional risk also demonstrated academic risk and vice versa. Specifically, 21 
students were identified as at-risk on academic performance and emotional well-being (either 
stress or school satisfaction), which corresponded to 34.4% of the 61 students at-risk on 
academic performance and 27.3% of the 77 students at-risk on either stress or school satisfaction. 
An examination of risk prevalence as a function of classroom teacher and student gender 
revealed relatively comparable base rates of any risk. Specifically, 25.0%, 35.9%, 36.6%, 39.8%, 
and 44.0% of students across sections taught by a given teacher were at-risk in at least one area. 
Also, 36.3% of boys and 40.0% of girls were at-risk in at least one area. 

Student report of academic performance. The correlation between values obtained 
from student report and school records was strong for both unweighted fall semester GPA (r = 
.74, p < .001) and end-of-semester grade in the specified AP or IB course (r= .85, p < .001). 
However, there were sizable numbers of inconsistencies around the cut points used by schools to 
identify risk. With respect to GPA, only 11 of 23 students (47.8%) whose school records 
indicated earned unweighted fall semester GPA < 3.0 also self-reported a GPA < 3.0, and an 
additional 10 students underestimated their GPA as < 3.0 when in fact school records indicated 
GPA > 3.0. With respect to AP/IB course grade, 35 of the 55 students (63.6%) whose school 
records indicated a grade of C, D, or F in the designated AP/IB course also self-reported a grade 


of C or below, and an additional 5 students underestimated their grade as C or below when in 
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fact school records indicated a grade of A or B. In sum, between 36.4% (n = 20) and 52.2% (n= 
12) of students with academic risk would be missed, and 2.0% (n = 5) to 3.6% (n = 10) of 
students without academic risk would be misidentified, if student self-report of AP/IB course 
grade and GPA, respectively, was used instead of gathering information on academic 
performance indicators from school records. For the remainder of this study, all analyses of 
academic performance utilized data from school records rather than self-report. 
Bivariate Relationships between Indicators 

Table 2 presents correlations between the indicators of student success (when analyzed in 
their continuous form) in the emotional and academic domains, and with teacher nominations (0 
= not at-risk; 1 = yes at-risk). The correlation between the two aspects of emotional well-being 
was moderate (r = -.31). The correlations between academic and emotional risk indicators were 
small in magnitude and not statistically significant (r = -.19 between GPA and stress). Teacher 
nominations were correlated in the expected directions with academic risk, both GPA (r= -.35) 
and course grade (r = -.45), but not significantly related to either indicator of emotional risk. 
Teacher Identification of Students with Any Emotional or Academic Risk 

True positive rate. As displayed in Table 3, 46 of the 117 students with any emotional or 
academic risk were nominated by their teachers as at-risk. There was a significant effect of 
teacher on proportion of students with any academic or emotional risk who were nominated as 
at-risk, y2 = 29.49, p < .001. As shown in Table 3, sensitivity rates were relatively low for 2 
teachers (3.1% and 26.7%) as compared to 57.1%, 59.5%, and 60.0% for the remaining three 
teachers. The average sensitivity rate was 35.7%, and a 95% confidence interval yielded a lower 
limit of 7.3% and an upper limit of 79.7%. The wide confidence interval is reflective of the 


uncertainty that stems from a small sample of teachers, who varied substantially in their 
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individual sensitivity. 

False negative rate. Seventy-one of the 117 students who demonstrated risk in terms of 
stress, engagement, or academic performance were not nominated by their teachers as at-risk. 
The average miss rate was 64.3% (95% CI = 20.3 - 92.7). 

True negative rate. One hundred fifty-seven of the 187 students without emotional or 
academic risk were not nominated by their teachers as at-risk. There was not a significant effect 
of teacher on proportion of students without any academic or emotional risk who were not 
nominated as at-risk, y2 = 7.86, p = .10. As shown in Table 3, all teachers’ true negative rate 
ranged from 66.7% to 95.0%, with a median of 80.8%. The average specificity rate was 83.5%, 
Thus, teachers correctly identified approximately 84% of students who did not have risk per 
rating scale or school records by intentionally not nominating them (95% CI = 64.4 — 93.4). 

False positive rate. Thirty of the 187 students whose self-reports and grades 
corresponded to low risk were identified by their teachers as at-risk. The average misidentified 
rate was 16.5% (95% CI = 6.6 — 35.6). 

Teacher Identification of Students with Elevated Stress 

True positive rate. As shown in Table 3, 15 of the 46 students who reported elevated 
stress were also nominated by their teachers as demonstrating risk. There was a significant effect 
of teacher on proportion of students with stress who were nominated as at-risk, y2 = 19.80, p = 
0005. A review of the teacher-specific proportions presented in Table 3 illustrates that 
sensitivity rates ranged from 5.6% to 100% across teachers, with a median of 40.0% true 
positives. The average sensitivity rate was 39.6%. A 95% confidence interval yielded a lower 
limit of 4.7% and an upper limit of 89.8%. 


False negative rate. Thirty-one of the 46 students who reported elevated stress levels 
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were not nominated by their teachers as demonstrating risk. The average miss rate was 60.4% 
(95% CI = 10.2 — 95.1). 
Teacher Identification of Students with Low School Satisfaction 

True positive rate. As summarized in Table 3, 14 of the 49 students who reported low 
school satisfaction were also nominated by their teachers as demonstrating risk. There was not a 
significant effect of teacher on proportion of students with at-risk school satisfaction who were 
nominated as at-risk, y2 = 7.48, p = .11. As shown in Table 3, no teacher achieved a true positive 
rate greater than 50%, with a median of 36.4% true positives. The average sensitivity was 27.9% 
(95% CI = 7.1 — 66.2). 

False negative rate. Thirty-five of the 49 students who reported low school satisfaction 
were not nominated by their teachers as demonstrating risk. The average false negative rate was 
72.1% (95% CI = 33.8 — 92.9). 

Teacher Identification of Students with Low Academic Performance 

True positive rate. As summarized in Table 3, 37 of the 61 students whose school 
records indicated grades below expectations were also nominated by their teachers as 
demonstrating risk. There was a significant effect of teacher on proportion of students with at- 
risk academic performance who were nominated as at-risk, y2 = 22.63, p < .001. As shown in 
Table 3, one teacher demonstrated particularly low sensitivity to academic risk (7.1% true 
positive) whereas the remaining four correctly identified between 66.7% and 100% of their 
students who were at-risk due to academic performance. The average true positive rate was 
59.9% (95% CI = 12.7 — 93.8). 

False negative rate. Twenty-four of the 61 students with academic risk were not 


nominated by their teachers as at-risk. The average miss rate was 40.1% (95% CI = 6.1 — 87.3). 
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Post-Hoc Analyses 

Additional analyses were conducted to determine whether students who emerged as at- 
risk on student report of stress and school satisfaction or report cards but were not identified by 
their teachers as at-risk differed from their counterparts who were accurately identified based on 
student characteristics (i.e., severity of emotional or academic risk, student gender). To 
determine if the true positive rate (and miss rate) depended on student characteristics, 
generalized mixed models were estimated, which included the student characteristics one at a 
time, while accounting for the nesting of students within teachers. 

Risk: Any emotional or academic indicator. Teachers were more likely to nominate 
students who demonstrated risk, and thus less likely to miss students, when the student received 
a lower grade in AP Human Geology or IB Biology, t (115) = -4.22, p < .001. For example, 
among the students who had demonstrated risk, the predicted probability of nomination for 
students with a grade of C was .61, whereas the predicted probability for students with a grade of 
B was .25. Similarly, teachers were less likely to miss students when the student had lower 
GPAs, t (115) = -3.21, p = .0017. For example, among the students who had demonstrated risk, 
the predicted probability of nomination for students with a GPA of 2.0 was .87, whereas the 
predicted probability for students with a GPA of 3.0 was .51. In addition, teachers were less 
likely to miss students when the students had a higher level of school satisfaction, ¢ (115) = 2.54, 
p = .0125. For example, among the students who had demonstrated risk, the predicted probability 
of nomination for students with a MSLSS score of 3 was .24, whereas the predicted probability 
for students with a MSLSS score of 5 was .53. No significant difference was found based on the 
student’s gender, f (115) = -1.18, p = .24, or level of perceived stress, t (115) = -0.77, p = .44. 


Risk: Stress. Students with stress were more likely to be nominated by teachers when 
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their GPA was lower, t (44) = -2.57, p = .0135 (predicted probability of nomination for students 
with GPAs of 2 and 3 were .99 and .73, respectively) and when the student had a low grade, t 
(44) = -2.59, p = .0128 (predicted probability of nomination for students with grades of C and B 
were .77 and .37, respectively). No significant differences in the probability of nomination were 
found based on the student’s gender, t (44) = 1.03, p = .31, perceived stress, t (44) = -0.29, p = 
.78, or school satisfaction, t (44) = -0.24, p = .81. 

Risk: School satisfaction. Students with at-risk school satisfaction were more likely to 
be nominated by teachers when their GPA was lower, t (47) = -2.67, p = .010 (predicted 
probability of nomination for students with GPAs of 2 and 3 were .96 and .56, respectively) and 
when the student had a low grade, t (44) = -2.83, p = .007 (predicted probability of nomination 
for students with a grades of C and B were .76 and .22, respectively). No significant differences 
in the probability of nomination were found based on the student’s gender, f (47) = -1.39, p = 
.17, perceived stress, t (47) = 0.49, p = .63, or school satisfaction, t (47) = 0.87, p = .39. 

Risk: Academic performance. The probability of teacher nomination of students with 
at-risk academic performance was not found to differ with student gender, tf (59) = -1.22, p = .23. 
Similarly, no significant differences were found for perceived stress, t (59) = 0.80, p = .43, 
school satisfaction, t (59) = 1.11, p= .27, GPA, t (59) = 1.24, p = .22, or course grade, t (59) = - 
0.94, p = .35. 

Discussion 

This study aimed to advance a multi-method, usable, and accessible approach for 
identifying AP/IB youth who demonstrate signs of academic or emotional challenges early in 
their high school career. The screening was embedded in a MTSS for student mental health that 


included a universal SEL curriculum intended to build 9 grade AP/IB students’ skills in stress 
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management and engagement. Following that Tier 1 classroom-based support, students showing 
signs of risk during the screening process were offered individualized, time-limited Tier 2 
support matched to the targets of the universal support and screening—stress/coping and student 
engagement. This study found that using student self-report of academic performance and 
teacher nomination of students at-risk would result in substantial miss rates of students identified 
as at risk emotionally or academically compared to student self-report of stress and school 
satisfaction, and academic performance data from school records. 
Prevalence of Risk among AP/IB Freshmen 

Base rates of emotional and academic risk among freshmen in accelerated curriculum are 
understudied. We first examined the proportion of students with risk on factors most salient to 
AP/IB student success, specifically perceived stress, affective engagement (school satisfaction), 
and academic performance in the first semester of high school. In general the 304 AP/IB students 
who participated in the screening process were functioning well across the three indicators, with 
the average student reporting s/he experiences mild satisfaction with school, and sometimes feels 
stressed. Academically they were also strong, with the average grade in AP Human Geography 
and IB Biology falling in the B range. Yet, approximately 39% of AP/IB freshmen who received 
Tier 1 services to develop social-emotional skills still showed signs of academic risk (GPA < 3.0, 
grade of C or lower in AP or IB class) or emotional risk (elevated stress, negative feelings about 
school). Across the risk factors, 15% of AP/IB students screened were at-risk for high stress, 
16% for low school satisfaction, and 20% for below par course grades. These prevalence rates of 
risk were on target with hypotheses (e.g., 15 — 20% of students identified per risk factor), but the 
low associations across types of risk led to a higher overall identification rate than one might 


expect among a group of students who took part in a research-based universal support with core 
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instruction in the targets relevant to the specific student population. 

As expected within a 3-tier prevention framework, the universal screening process 
identified a sizeable proportion of AP/IB students with signs of risk, thus a subset of students to 
be further considered for early intervention. The relatively high total percentage (1.e., 38.5% of 
freshmen with at least one risk factor, in comparison to a more typically expected 20% of 
students in need of more intensive supports beyond the Tier | curriculum) may be in part a 
function of the diverse targets assessed and potentially liberal cut scores for “risk” used in this 
study. While the screening process might yield a base rate of risk that could be challenging to 
serve with typical school resources (Kilgus & Eklund, 2016), the measures and cut scores are 
appropriate for normative levels of emotional and academic well-being in this unique population. 
Traditional school screening methods often measure symptoms of psychopathology rather than 
precursors of problems like stress stemming from academic demands and school satisfaction, 
both pertinent risk factors for AP/IB youth (Suldo et al., 2008; Suldo et al., 2018). Likewise, 
academic risk levels are considerably different among AP/IB youth as compared to their general 
education peers. Case in point, regulations for the award of the IB Diploma specify that students 
cannot have more than three grades of C (or lower) across their IB coursework (IBO, 2016). 
Whereas schools commonly flag a student for being at-risk (e.g., for school dropout) if they are 
failing a class, a C or lower is a better indicator of risk (e.g., for withdrawal from accelerated 
courses) for AP/IB students. Identification of 9th grade AP/IB students with early signs of 
emotional or academic risk provides an opportunity for Tier 2 supports to prevent further 
problems. Research on prevention services indicates it is more cost-effective to provide services 
before youth experience clinically significant crises as opposed to more intensive Tier 3 services 


(Cooney, Kratochwill, & Small, 2010). 
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Accuracy of Teacher Nominations 

With efficiency in mind, a primary aim of this study was to examine the accuracy of 
teacher nominations in identifying those students who demonstrated risk on emotional factors 
(stress, school satisfaction) or academic performance. Our results suggest teachers suspect fewer 
students experience signs of academic or emotional problems as 25% of the sample was 
nominated by teachers as at-risk whereas about 39% of students emerged as at-risk based on self- 
report of emotional indicators and school academic records. Related to their tendency to not 
over-nominate, teachers demonstrated relatively high accuracy in terms of specificity; 84% of 
students without elevated risk on academic or emotional indicators were also not indicated as at- 
risk by their teachers. Such specificity is helpful as to avoid unnecessary resource allocation to 
students who do not need extra supports. However, as nominated students are further considered 
for Tier 2 supports, they can be ruled out, if appropriate, as additional data become available. 

In any screening procedure, sensitivity is often weighted more heavily than specificity, 
given the high stakes involved in inadvertently excluding a student from further consideration for 
supports that may indeed be warranted. This study found that AP/IB teachers correctly identified 
only 39% of students with emotional or academic risk. If the teacher nomination method had 
been relied on as the sole way to identify students for Tier 2 supports, about 61% of students 
with elevated stress, low school satisfaction, or substandard academic performance would have 
been missed, particularly those students with emotional risk vs. academic risk. Kilgus and 
colleagues’ (2016) study of the accuracy of teacher nominations for elementary and middle 
school students also found lower sensitivity rates associated with emotional risk in comparison to 
academic or social risk, in part because teachers tended to nominate fewer students as showing 


emotional risk; sensitivity remained less than adequate even after they changed directions on the 
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nomination form to identify a minimum number of students in each risk category (i.e., emotional, 
academic, social). Similarly, an examination of the symptom profiles of high school students 
referred for intervention by school personnel due to “exhibiting the most severe behavioral, 
emotional, or school problems” (p. 100) concluded that educator nominations under-identify 
students with internalizing forms of problems (Splett, George, et al., 2018). 

The current study found teacher and student characteristics were related to the accuracy 
of the nomination procedure. Regarding teacher effects, we found significant variability among 
teachers in their sensitivity rates, with a given teacher identifying from 3% to 60% of his or her 
students who had any form of risk. Regarding student effects, at-risk students whose teachers 
nominated them as such were more likely to be students who had lower academic performance; 
students with elevated stress or low emotional engagement who had relatively higher grades 
were less likely to be on teachers’ radars for showing signs of emotional risk. These findings are 
similar to Splett, Trainor, and colleagues’ (2018) finding that traditional school referral 
mechanisms were more likely to identify students with lower course grades. In that comparison 
of elementary school students who were already receiving school-based interventions to students 
identified as at-risk through universal screening with rating scales, Splett et al. also found gender 
and symptom severity were associated with school identification of risk, with males and students 
with higher internalizing or externalizing problem scores being more likely to receive 
interventions. In contrast, the current study of high school students found males and students 
with especially high stress levels were not particularly likely to be nominated by teachers as 
demonstrating academic or emotional challenges. Rather than detecting the most symptomatic 
students, students with particularly low school satisfaction were less likely to be detected by 


teachers as demonstrating signs of risk, perhaps because student disaffection with school 
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precludes teachers from having sufficient familiarity with the student to detect problems. 

The associations between teacher nomination status and students’ continuous scores on 
indicators used to assess emotional and academic status revealed no relationship with student 
report of stress or school satisfaction, but moderate correlations in the expected direction with 
school grades. The tendency for AP/IB teachers to equate academic underachievement with risk 
is consistent with previous research conducted both with gifted and non-gifted students, finding 
that students with lower achievement may be more likely to be identified as at risk academically 
compared to students with higher academic indicators (Eklund & Dowdy, 2014). 

Regarding sensitivity to academic risk, teachers identified on average about 60% of 
students deemed at-risk due to academic performance (GPA < 3.0 or grade in a rigorous AP/IB 
course a “C” or below at the end of the first semester). The relatively high accuracy with which 
teachers identified students with below par academic performance is not surprising given 
teachers frequently collect and examine indicators of achievement. In spite of such familiarity 
with students’ academic performance in their class, sensitivity to academic risk varied 
significantly between teachers and only one teacher’s nominations were 100% in agreement with 
risk as determined by school records. Another teacher greatly underestimated the number of 
students in her class with academic challenges, identifying only 7% of students who ultimately 
met criteria for at-risk academic performance. 

In comparison to identifying academic risk, teachers in this study were less accurate in 
identifying those students in their AP/IB class(s) who were struggling emotionally, identifying 
on average only 28% of students with low affective engagement (school satisfaction) and 40% of 
students with elevated stress. These findings provide modest support for the sensitivity of teacher 


nominations in identifying high school students with internalizing forms of risk. The modest 
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sensitivity to low school satisfaction was consistent across teachers, with no teacher correctly 
identifying more than half of his/her students who actually reported negative feelings about 
school. In contrast, sensitivity to detection of student stress varied significantly across teacher, 
with two teachers with particularly low sensitivity (< 15%) and two teachers with particularly 
high sensitivity (> 78%). All teachers in this study were present for a SEL curriculum delivered 
in their classroom that explicitly focused on student stress and affective engagement, which may 
have facilitated the high sensitivity to student stress demonstrated by some teachers. Additional 
research is warranted to explore factors related to attitudes and skills that may contribute to 
instances of high miss rates. Many secondary teachers report low confidence in their own ability 
to accurately identify students with internalizing problems (Papandrea & Winefield, 2011), 
perhaps due to the emphasis on academic success in the classroom. The poor sensitivity of 
teacher nominations to detecting AP/IB students at risk due to low school satisfaction or elevated 
stress mirrors findings from prior studies that found teacher nomination procedures catch less 
than half of students who experience internalizing symptoms of mental health distress (Auger, 
2004; Cunningham & Suldo, 2014; Moor et al., 2007). Preservice training or professional 
development may include too little information on student affective needs, such as signs of stress 
or disconnect from school. 
Accuracy of Student Report of Academic Performance 

We hypothesized that AP/IB students would be able to accurately report their end-of- 
course grades, which might permit a more efficient screening process with fewer pieces of data 
needing to be gathered by the school. However, the results indicated that freshmen, especially 
those performing in the C range, are an unreliable source of academic information. If student 


self-report of AP/IB course grade and GPA had been relied on as the sole way to identify 
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students for further consideration for tiered supports, between 36% and 52% of students with 
academic risk (per actual report cards) would have been missed. Schwartz and Beaver (2015) 
similarly found that high school students tend to over-report their GPA—by about one-half letter 
grade on average—although inflated self-reported grades are more common among adolescents 
with lower achievement and cognitive ability. In the current study, GPA values from student 
self-report and school records were correlated reflecting that AP/IB students indeed have a 
general idea of their end of semester grades, but when it comes to making decisions based on 
dichotomized indicators that require accuracy around the cut point, gathering information from 
school records appears warranted regardless of the effort needed to obtain and organize the data. 
Limitations and Directions for Future Research 

Conclusions from this first study of different ways to systematically identify high school 
students in accelerated courses with early signs of academic or emotional risk are tempered by 
setting-specific features. Participants were drawn from only two high schools. Base rates of risk 
and accuracy of methods other than student report (of stress and school satisfaction) and school 
records (of academic performance) should be investigated in a larger sample of schools, 
programs, teachers, and students. Further, this screening study was embedded in a larger research 
project, in which all teachers participated in a universal SEL curricula that addressed the 
correlates of AP/IB student success, including stress management and student engagement. This 
Tier 1 support may have influenced how teachers understood and viewed student stress and 
affective engagement, perhaps contributing to an overestimation of the sensitivity of teacher 
nominations. Teachers without such exposure to social-emotional factors relevant to AP/IB 
student success may perform differently on similar screening procedures. Future studies might 


explore the extent to which teacher sensitivity to student risk increases as a function of (a) 
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exposure to affective needs of AP/IB students, (b) routine monitoring of student emotional status 
including internalizing behavior, and/or (c) familiarity with the nomination procedure. 

The misidentification rate (14% of students) yielded in this study suggests that teacher 
nominations might add unique information about the pool of students in need of further 
consideration by school teams. Future research might explore what led teachers to nominate 
students without high stress, low school satisfaction, or below par grades as at-risk for emotional 
or academic challenges. Perhaps knowledge of a student’s family situation or other 
environmental stressor led a teacher to feel a student warrants consideration for targeted supports 
(even if outside the scope of planned supports geared toward AP/IB student success). Adding 
teacher nominations to a multi-method screening that also includes data from students and 
records might also be beneficial to increasing teacher buy-in to the overall screening process. 

In this study, it was not possible to calculate all possible conditional probability indices 
because the teacher nomination form did not ask teachers to evaluate students for challenges in 
discrete areas (e.g., stress, school, academic performance) separately. Future studies might 
consider asking teachers to nominate which students demonstrate signs of emotional risk, 
academic risk, or both. Additionally, we involved teachers, administrators, and school 
psychologists—but not students—in the creation of the screening procedures, such as the content 
and structure of the educator nomination form and selection of academic course to monitor. 
Future research could explore student acceptability of the targets examined, data sources used, 
and overarching purpose (e.g., link to Tier 2 support) within a mid-year screening. In this study, 
students were informed of the purpose of screening before being asked to complete the survey, 
but the research staff did not query the validity of students’ specific responses. In contrast, 


researcher bias could have been introduced into teachers’ nomination accuracy by encouraging 
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the teacher with a particularly low nomination base rate to reconsider her class roster. 
Conclusion and Implications 

Grounded in evidence-based practices and data-based decision making, MTSS 
emphasizes prevention, screening, and early intervention so that students may receive 
appropriate services before they have the chance to fail (Sailor, Doolittle, Bradley, & Danielson, 
2009). The methods described and evaluated in this study illustrate how to assess specific risk 
factors that predict diminished success, in order to identify AP/IB freshmen for targeted support. 
Results provide preliminary support for a screening procedure that incorporates data from (a) 
student self-report of stress and affective engagement (school satisfaction), and (b) school 
records of first semester GPA and grade in an AP or IB course that discriminates satisfactory 
from below par performance in accelerated curricula. Given the sizeable proportions of at-risk 
students missed in the teacher nomination procedure, especially those with emotional challenges 
(high stress, low school satisfaction), findings do not lend strong support for bypassing the 
collection of data from students when the construct of interest is internalizing in nature. Also, 
findings support the need to collect additional data on academic performance from school 
records, as students were only moderately accurate in self-reporting grades. The effort needed to 
gather data from multiple methods is justified by the ability to offer targeted supports to all 


AP/IB students with emotional and academic needs early in their high school career. 
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Table 1 


Proportion of Freshmen in Advanced Placement and International Baccalaureate Courses 
with Risk on Dichotomized Emotional and Academic Indicator (N = 304) 


Risk Indicator (Dichotomized) N % N % 
Stress (PSS > 3.6) 258 84.87 46 15.13 
School Satisfaction (MSLSS < 3.4) 25) 83.88 49 16.12 
Academic Performance 243 79.90 61 20.10 
GPA (< 3.0) 281 92.43 23 47 
AP/IB Course Grade (C, D, or F) 249 81.91 55 18.09 


Note. GPA = unweighted semester GPA. PSS = Perceived Stress Scale. MSLSS = 
Multidimensional Students Life Satisfaction Scale. AP = Advanced Placement. IB = 


International Baccalaureate. 
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Table 2 


Correlations between Indicators of Student Risk (Continuous Scores) and Teacher 
Nomination Status 


Indicator Teacher Stress School Semester GPA 
Nomination Satisfaction 

Stress 02 

School Satisfaction -.03 -.31** 

Semester GPA -.35** -. 19** 11 

Course Grade - 45** -.09 05 .76** 


Note. GPA = unweighted semester GPA. Stress, school satisfaction, GPA, and 
course grade were continuously scaled. Teacher nomination variable coded as 0 = 
not at-risk, 1 = yes at-risk. 

* p< .05, **p < 01 
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Table 3 
Accuracy of Teachers in Identifying Students with Emotional or Academic Risk 
Risk Indicator (Dichotomized) Sensitivity Specificity PPV NPV 
Any Risk Total (V = 304) 39:32 83.96 60.53 68.86 
Female (N = 180) 44.44 84.26 65.31 69.47 
Teacher 1 (N = 72) oe13 95.00 33733 55.07 
Teacher 2 (N = 20) 60.00 66.67 37.50 83.33 
Teacher 3 (N = 41) 26.67 80.77 44.44 65.63 
Teacher 4 (N = 93) 59.46 85.71 fies) 76.19 
Teacher 5 (N = 78) 57.14 80.00 61.54 76.92 
Stress Total (N = 304) 32.61 
Female (N = 180) 31.43 
Teacher | (N = 72) 5.56 
Teacher 2 (N = 20) 100.00 
Teacher 3 (N = 41) 14.29 
Teacher 4 (N = 93) 40.00 
Teacher 5 (N = 78) 77.78 
School Satisfaction Total (NV = 304) 28.57 
Female (N = 180) 37.04 
Teacher 1 (N = 72) 0.00 
Teacher 2 (N = 20) 50.00 
Teacher 3 (N = 41) 25.00 
Teacher 4 (N = 93) 36.36 
Teacher 5 (N = 78) 42.86 
Academic Risk Total (NV = 304) 60.66 
Female (N = 180) 66.67 
Teacher | (N = 72) 7.14 
Teacher 2 (N = 20) 100.00 
Teacher 3 (N = 41) 66.67 
Teacher 4 (N = 93) 80.77 
Teacher 5 (N = 78) 71.43 


Note. PPV = Positive Predictive Value. NPV = Negative Predictive Value. Female refers to 


student gender. 
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Appendix: Educator Nomination Form 


We are identifying students with signs of academic or emotional risk. Those students will be 
offered additional supports intended to help them be successful in AP/IB. 


Directions: Please review the attached list of eligible 9tn grade students in your AP/IB class. 
Then, identify the participating students that, based on your knowledge of this student and 
his/her typical behavior, demonstrate academic or emotional challenges in AP/IB. Example 
student behaviors that may indicate academic or emotional risk are listed below. You may check 
“ves” for as few or as many students as you feel fit the criteria below for being at-risk for 
diminished success in AP/IB. 


Please do not discuss your thoughts on this list with any colleagues; please complete this form 
independently by checking “yes,” or circling the names, for the students who are demonstrating 
academic or emotional challenges. Thank you! 


At-Risk for Diminished Success in AP/IB: 
Examples of Signs of Academic or Emotional Challenges in AP/IB 


Poor test, quiz, and exam grades Appears burnt out on schoolwork 


Does not turn in assignments on time (may Seems unhappy during class (e.g., tearful) 


make frequent requests for extended time) 


Seems disinterested during class Makes negative statements about AP/IB or 
school 

Difficulty coping effectively with academic Appears lonely or socially isolated (no friends 

demands in AP/TB) 

Gives up or stops trying on schoolwork Expresses extreme or frequent worry about 


performance on assignments or exams 


Expresses frequent or extreme self-doubt Complains excessively about workload or 


about ability to achieve in AP/IB particular assignments 


Does not seem to take schoolwork seriously Misses class (e.g., signs in and out of school, 
(e.g., plays around during class) skips school, stays in bathroom during class) 


Other: 


